Problem Statement¶
Context¶
Businesses like banks which provide service have to worry about problem of 'Customer Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on improvement of service, keeping in mind these priorities.
Objective¶
You as a Data scientist with the bank need to build a neural network based classifier that can determine whether a customer will leave the bank or not in the next 6 months.
Data Dictionary¶
CustomerId: Unique ID which is assigned to each customer
Surname: Last name of the customer
CreditScore: It defines the credit history of the customer.
Geography: A customer’s location
Gender: It defines the Gender of the customer
Age: Age of the customer
Tenure: Number of years for which the customer has been with the bank
NumOfProducts: refers to the number of products that a customer has purchased through the bank.
Balance: Account balance
HasCrCard: It is a categorical variable which decides whether the customer has credit card or not.
EstimatedSalary: Estimated salary
isActiveMember: Is is a categorical variable which decides whether the customer is active member of the bank or not ( Active member in the sense, using bank products regularly, making transactions etc )
Exited : whether or not the customer left the bank within six month. It can take two values
** 0=No ( Customer did not leave the bank ) ** 1=Yes ( Customer left the bank )
Importing necessary libraries¶
In [432]:
!pip install tensorflow==2.18.0 scikit-learn==1.3.2 matplotlib===3.8.0 seaborn==0.13.1 numpy==1.26.4 pandas==2.2.2 -q --user
In [433]:
# Imblearn libary is used to handle imbalanced data
!pip install imblearn --user
!pip install imbalanced-learn --user
Requirement already satisfied: imblearn in /root/.local/lib/python3.11/site-packages (0.0) Requirement already satisfied: imbalanced-learn in /usr/local/lib/python3.11/dist-packages (from imblearn) (0.13.0) Requirement already satisfied: numpy<3,>=1.24.3 in /usr/local/lib/python3.11/dist-packages (from imbalanced-learn->imblearn) (1.26.4) Requirement already satisfied: scipy<2,>=1.10.1 in /usr/local/lib/python3.11/dist-packages (from imbalanced-learn->imblearn) (1.13.1) Requirement already satisfied: scikit-learn<2,>=1.3.2 in /root/.local/lib/python3.11/site-packages (from imbalanced-learn->imblearn) (1.3.2) Requirement already satisfied: sklearn-compat<1,>=0.1 in /usr/local/lib/python3.11/dist-packages (from imbalanced-learn->imblearn) (0.1.3) Requirement already satisfied: joblib<2,>=1.1.1 in /usr/local/lib/python3.11/dist-packages (from imbalanced-learn->imblearn) (1.4.2) Requirement already satisfied: threadpoolctl<4,>=2.0.0 in /usr/local/lib/python3.11/dist-packages (from imbalanced-learn->imblearn) (3.5.0) Requirement already satisfied: imbalanced-learn in /usr/local/lib/python3.11/dist-packages (0.13.0) Requirement already satisfied: numpy<3,>=1.24.3 in /usr/local/lib/python3.11/dist-packages (from imbalanced-learn) (1.26.4) Requirement already satisfied: scipy<2,>=1.10.1 in /usr/local/lib/python3.11/dist-packages (from imbalanced-learn) (1.13.1) Requirement already satisfied: scikit-learn<2,>=1.3.2 in /root/.local/lib/python3.11/site-packages (from imbalanced-learn) (1.3.2) Requirement already satisfied: sklearn-compat<1,>=0.1 in /usr/local/lib/python3.11/dist-packages (from imbalanced-learn) (0.1.3) Requirement already satisfied: joblib<2,>=1.1.1 in /usr/local/lib/python3.11/dist-packages (from imbalanced-learn) (1.4.2) Requirement already satisfied: threadpoolctl<4,>=2.0.0 in /usr/local/lib/python3.11/dist-packages (from imbalanced-learn) (3.5.0)
In [434]:
# Library for data manipulation and analysis.
import pandas as pd
# Fundamental package for scientific computing.
import numpy as np
#splitting datasets into training and testing sets.
from sklearn.model_selection import train_test_split
#Imports tools for data preprocessing including label encoding, one-hot encoding, and standard scaling
from sklearn.preprocessing import LabelEncoder, OneHotEncoder,StandardScaler
#Imports a class for imputing missing values in datasets.
from sklearn.impute import SimpleImputer
#Imports the Matplotlib library for creating visualizations.
import matplotlib.pyplot as plt
# Imports the Seaborn library for statistical data visualization.
import seaborn as sns
# Time related functions.
import time
#Imports functions for evaluating the performance of machine learning models
from sklearn.metrics import confusion_matrix, f1_score,accuracy_score, recall_score, precision_score, classification_report
#Imports the tensorflow,keras and layers.
import tensorflow
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Dense, Input, Dropout,BatchNormalization
from tensorflow.keras import backend
# To oversample the data
from imblearn.over_sampling import SMOTE
# to suppress unnecessary warnings
import warnings
warnings.filterwarnings("ignore")
In [435]:
# Set the seed using keras.utils.set_random_seed. This will set:
# 1) `numpy` seed
# 2) backend random seed
# 3) `python` random seed
tf.keras.utils.set_random_seed(812)
# If using TensorFlow, this will make GPU ops as deterministic as possible,
# but it will affect the overall performance, so be mindful of that.
tf.config.experimental.enable_op_determinism()
Loading the dataset¶
In [436]:
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
In [437]:
url = "/content/drive/My Drive/Colab Notebooks/Neural Networks/Project/bank.csv"
Bank_df = pd.read_csv(url)
In [438]:
data_df = Bank_df.copy()
Data Overview¶
In [439]:
Bank_df.head()
Out[439]:
| RowNumber | CustomerId | Surname | CreditScore | Geography | Gender | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 15634602 | Hargrave | 619 | France | Female | 42 | 2 | 0.00 | 1 | 1 | 1 | 101348.88 | 1 |
| 1 | 2 | 15647311 | Hill | 608 | Spain | Female | 41 | 1 | 83807.86 | 1 | 0 | 1 | 112542.58 | 0 |
| 2 | 3 | 15619304 | Onio | 502 | France | Female | 42 | 8 | 159660.80 | 3 | 1 | 0 | 113931.57 | 1 |
| 3 | 4 | 15701354 | Boni | 699 | France | Female | 39 | 1 | 0.00 | 2 | 0 | 0 | 93826.63 | 0 |
| 4 | 5 | 15737888 | Mitchell | 850 | Spain | Female | 43 | 2 | 125510.82 | 1 | 1 | 1 | 79084.10 | 0 |
In [440]:
Bank_df.shape
Out[440]:
(10000, 14)
Observations:
- There are 10,000 rows and 14 columns.
In [441]:
Bank_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 14 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 RowNumber 10000 non-null int64 1 CustomerId 10000 non-null int64 2 Surname 10000 non-null object 3 CreditScore 10000 non-null int64 4 Geography 10000 non-null object 5 Gender 10000 non-null object 6 Age 10000 non-null int64 7 Tenure 10000 non-null int64 8 Balance 10000 non-null float64 9 NumOfProducts 10000 non-null int64 10 HasCrCard 10000 non-null int64 11 IsActiveMember 10000 non-null int64 12 EstimatedSalary 10000 non-null float64 13 Exited 10000 non-null int64 dtypes: float64(2), int64(9), object(3) memory usage: 1.1+ MB
Observations:
- There are 11 numerical columns and 3 object column.
In [442]:
Bank_df.describe().T
Out[442]:
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| RowNumber | 10000.0 | 5.000500e+03 | 2886.895680 | 1.00 | 2500.75 | 5.000500e+03 | 7.500250e+03 | 10000.00 |
| CustomerId | 10000.0 | 1.569094e+07 | 71936.186123 | 15565701.00 | 15628528.25 | 1.569074e+07 | 1.575323e+07 | 15815690.00 |
| CreditScore | 10000.0 | 6.505288e+02 | 96.653299 | 350.00 | 584.00 | 6.520000e+02 | 7.180000e+02 | 850.00 |
| Age | 10000.0 | 3.892180e+01 | 10.487806 | 18.00 | 32.00 | 3.700000e+01 | 4.400000e+01 | 92.00 |
| Tenure | 10000.0 | 5.012800e+00 | 2.892174 | 0.00 | 3.00 | 5.000000e+00 | 7.000000e+00 | 10.00 |
| Balance | 10000.0 | 7.648589e+04 | 62397.405202 | 0.00 | 0.00 | 9.719854e+04 | 1.276442e+05 | 250898.09 |
| NumOfProducts | 10000.0 | 1.530200e+00 | 0.581654 | 1.00 | 1.00 | 1.000000e+00 | 2.000000e+00 | 4.00 |
| HasCrCard | 10000.0 | 7.055000e-01 | 0.455840 | 0.00 | 0.00 | 1.000000e+00 | 1.000000e+00 | 1.00 |
| IsActiveMember | 10000.0 | 5.151000e-01 | 0.499797 | 0.00 | 0.00 | 1.000000e+00 | 1.000000e+00 | 1.00 |
| EstimatedSalary | 10000.0 | 1.000902e+05 | 57510.492818 | 11.58 | 51002.11 | 1.001939e+05 | 1.493882e+05 | 199992.48 |
| Exited | 10000.0 | 2.037000e-01 | 0.402769 | 0.00 | 0.00 | 0.000000e+00 | 0.000000e+00 | 1.00 |
Observations:
RowNumber - unique number - Ignore
CustomerID - unique customer identifier - Ignore
CreditScore - Mean is 650.528 and Median is 652
Age - Mean is 38.92 years and Median is 37 years
Tenure - Mean is 5.01 and Median is 5
Balance - Mean is 76485 and Median is 97198
NumOfProducts - Mean is 1.53 and Median is 1
HasCrCard - Mean is 0.705 and Median is 1
IsActiveMember - Mean is 0.5 and Median is 1
EstimatedSalary - Mean is 100090.2 and Median is 100193
Exited - 0 and 1 are the values and is the target variable
In [443]:
Bank_df.isnull().sum()
Out[443]:
| 0 | |
|---|---|
| RowNumber | 0 |
| CustomerId | 0 |
| Surname | 0 |
| CreditScore | 0 |
| Geography | 0 |
| Gender | 0 |
| Age | 0 |
| Tenure | 0 |
| Balance | 0 |
| NumOfProducts | 0 |
| HasCrCard | 0 |
| IsActiveMember | 0 |
| EstimatedSalary | 0 |
| Exited | 0 |
Observations:
- There are no null values.
In [444]:
Bank_df.duplicated().sum()
Out[444]:
0
Observations:
- There are no duplicates.
In [445]:
Bank_df['Exited'].value_counts()
Out[445]:
| count | |
|---|---|
| Exited | |
| 0 | 7963 |
| 1 | 2037 |
In [446]:
Bank_df['Exited'].value_counts(normalize=True)
Out[446]:
| proportion | |
|---|---|
| Exited | |
| 0 | 0.7963 |
| 1 | 0.2037 |
Observations:
- There are 79.63% customers who did not leave the bank.
- There are 20.37% customers who left the bank.
- This is an imbalanced dataset.
Removing features with only unique values
In [447]:
Bank_df.nunique()
Out[447]:
| 0 | |
|---|---|
| RowNumber | 10000 |
| CustomerId | 10000 |
| Surname | 2932 |
| CreditScore | 460 |
| Geography | 3 |
| Gender | 2 |
| Age | 70 |
| Tenure | 11 |
| Balance | 6382 |
| NumOfProducts | 4 |
| HasCrCard | 2 |
| IsActiveMember | 2 |
| EstimatedSalary | 9999 |
| Exited | 2 |
Observations:
- Remove RowNumber and CustomerId since they are unique identifiers.
- Keep EstimatedSalary since it might be a important feature which will influence financial decisions and Exiting/not Exiting likelihood.
In [448]:
#remove RowNumber
Bank_df.drop(['RowNumber'], axis=1, inplace=True)
In [449]:
#remove CustomerID
Bank_df.drop(['CustomerId'], axis=1, inplace=True)
In [450]:
Bank_df.head()
Out[450]:
| Surname | CreditScore | Geography | Gender | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Hargrave | 619 | France | Female | 42 | 2 | 0.00 | 1 | 1 | 1 | 101348.88 | 1 |
| 1 | Hill | 608 | Spain | Female | 41 | 1 | 83807.86 | 1 | 0 | 1 | 112542.58 | 0 |
| 2 | Onio | 502 | France | Female | 42 | 8 | 159660.80 | 3 | 1 | 0 | 113931.57 | 1 |
| 3 | Boni | 699 | France | Female | 39 | 1 | 0.00 | 2 | 0 | 0 | 93826.63 | 0 |
| 4 | Mitchell | 850 | Spain | Female | 43 | 2 | 125510.82 | 1 | 1 | 1 | 79084.10 | 0 |
Exploratory Data Analysis¶
Univariate Analysis¶
Categorical Variables : (Barplot on the below variables)
Geography, Gender
Numerical Variables : (Histplot and Boxplot on the below variables)
CreditScore, Age, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, Exited
In [451]:
# function to plot a boxplot and a histogram along the same scale.
def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
"""
Boxplot and histogram combined
data: dataframe
feature: dataframe column
figsize: size of figure (default (12,7))
kde: whether to the show density curve (default False)
bins: number of bins for histogram (default None)
"""
f2, (ax_box2, ax_hist2) = plt.subplots(
nrows=2, # Number of rows of the subplot grid= 2
sharex=True, # x-axis will be shared among all subplots
gridspec_kw={"height_ratios": (0.25, 0.75)},
figsize=figsize,
) # creating the 2 subplots
sns.boxplot(
data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
) # boxplot will be created and a triangle will indicate the mean value of the column
sns.histplot(
data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
) if bins else sns.histplot(
data=data, x=feature, kde=kde, ax=ax_hist2
) # For histogram
ax_hist2.axvline(
data[feature].mean(), color="green", linestyle="--"
) # Add mean to the histogram
ax_hist2.axvline(
data[feature].median(), color="black", linestyle="-"
) # Add median to the histogram
Observation on CreditScore
In [452]:
histogram_boxplot(Bank_df, 'CreditScore')
Observartions:
- CreditScore a left skewed distribution
- Mean and Median are pretty close and similar.
- There are outliers in the data.
Observation on Age
In [453]:
histogram_boxplot(Bank_df, 'Age')
Observations:
- Age is right skewed distribution
- There are outliers
Observation on Tenure
In [454]:
histogram_boxplot(Bank_df, 'Tenure')
Observations:
- Distribution is uniform
- There are no outliers
- Mean and median are around the same
Observation on Balance
In [455]:
histogram_boxplot(Bank_df, 'Balance')
Observations:
- There are lot of customers(around 3600) with 0 balance
- distribution is right skewed
Observation on NumOfProducts
In [456]:
histogram_boxplot(Bank_df, 'NumOfProducts')
Observations:
- there are around 5000 customers using 1 product followed by customers who use 2 products
- There are few customers who use 3 and 4
- There is outlier
Observation on HasCrCard
In [457]:
histogram_boxplot(Bank_df, 'HasCrCard')
Observations:
- There are around 7000 customers who have credit card and around 3000 customers who dont have credit card.
Observation on IsActiveMember
In [458]:
histogram_boxplot(Bank_df, 'IsActiveMember')
Observations:
- Around 5100 customers are active customers and 4900 customers are inactive customers
Observation on EstimatedSalary
In [459]:
histogram_boxplot(Bank_df, 'EstimatedSalary')
Observations:
- Estimated salary is uniform distribution
- Mean and Median are similar.
Observation on Exited
In [460]:
histogram_boxplot(Bank_df, 'Exited')
Observations:
- There are around 8000 customer who are with the bank and around 2000 customers who have exited.
In [461]:
# function to create labeled barplots
def labeled_barplot(data, feature, perc=False, n=None):
"""
Barplot with percentage at the top
data: dataframe
feature: dataframe column
perc: whether to display percentages instead of count (default is False)
n: displays the top n category levels (default is None, i.e., display all levels)
"""
total = len(data[feature]) # length of the column
count = data[feature].nunique()
if n is None:
plt.figure(figsize=(count + 1, 5))
else:
plt.figure(figsize=(n + 1, 5))
plt.xticks(rotation=90, fontsize=15)
ax = sns.countplot(
data=data,
x=feature,
palette="Paired",
order=data[feature].value_counts().index[:n].sort_values(),
)
for p in ax.patches:
if perc == True:
label = "{:.1f}%".format(
100 * p.get_height() / total
) # percentage of each class of the category
else:
label = p.get_height() # count of each level of the category
x = p.get_x() + p.get_width() / 2 # width of the plot
y = p.get_height() # height of the plot
ax.annotate(
label,
(x, y),
ha="center",
va="center",
size=12,
xytext=(0, 5),
textcoords="offset points",
) # annotate the percentage
plt.show() # show the plot
Observation on Geography
In [462]:
labeled_barplot(Bank_df, 'Geography')
Observations:
- Customers from France are the highest followed by Germany and Spain
Observation on Gender
In [463]:
labeled_barplot(Bank_df, 'Gender')
Observations:
- Highest number of customers are male.
Bivariate Analysis¶
In [464]:
sns.pairplot(data=Bank_df,hue='Exited')
Out[464]:
<seaborn.axisgrid.PairGrid at 0x7ad8452f8810>
Observations:
- We do not see positive or negative correlation from pairplot.
In [465]:
plt.figure(figsize=(15, 7))
numeric_df = Bank_df.select_dtypes(include=['number'])
sns.heatmap(numeric_df.corr(), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")
plt.show()
Observations:
- From heatmap we observe some postive and negative correlation although none of them are highly correlated.
Positive correlation:
- Age and Exited
Negative Correlation:
- Balance and NumOfProducts
Analysis of relationships of all variables on target variable¶
Relationship of categorical variables on Target variable
Geography vs Exited
Gender vs Exited
In [466]:
# function to plot stacked bar chart
def stacked_barplot(data, predictor, target):
"""
Print the category counts and plot a stacked bar chart
data: dataframe
predictor: independent variable
target: target variable
"""
count = data[predictor].nunique()
sorter = data[target].value_counts().index[-1]
tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
by=sorter, ascending=False
)
print(tab1)
print("-" * 120)
tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
by=sorter, ascending=False
)
tab.plot(kind="bar", stacked=True, figsize=(count + 1, 5))
plt.legend(
loc="lower left", frameon=False,
)
plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
plt.show()
Geography vs Exited
In [467]:
stacked_barplot(Bank_df, 'Geography', 'Exited')
Exited 0 1 All Geography All 7963 2037 10000 Germany 1695 814 2509 France 4204 810 5014 Spain 2064 413 2477 ------------------------------------------------------------------------------------------------------------------------
Observations:
- There is more customer churn in Germany.Exited customers distribution in Spain and France are very similar
Gender vs Exited
In [468]:
stacked_barplot(Bank_df, 'Gender', 'Exited')
Exited 0 1 All Gender All 7963 2037 10000 Female 3404 1139 4543 Male 4559 898 5457 ------------------------------------------------------------------------------------------------------------------------
Observations:
- More Female customers have churned.
Relationship of numerical variables on target variable
CreditScore vs Exited
Age vs Exited
Tenure vs Exited
Balance vs Exited
NumOfProducts vs Exited
HasCrCard vs Exited
IsActiveMember vs Exited
EstimatedSalary vs Exited
In [469]:
### Function to plot distributions
def distribution_plot_wrt_target(data, predictor, target):
fig, axs = plt.subplots(2, 2, figsize=(12, 10))
target_uniq = data[target].unique()
axs[0, 0].set_title("Distribution of target for target=" + str(target_uniq[0]))
sns.histplot(
data=data[data[target] == target_uniq[0]],
x=predictor,
kde=True,
ax=axs[0, 0],
color="teal",
)
axs[0, 1].set_title("Distribution of target for target=" + str(target_uniq[1]))
sns.histplot(
data=data[data[target] == target_uniq[1]],
x=predictor,
kde=True,
ax=axs[0, 1],
color="orange",
)
axs[1, 0].set_title("Boxplot w.r.t target")
sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
axs[1, 1].set_title("Boxplot (without outliers) w.r.t target")
sns.boxplot(
data=data,
x=target,
y=predictor,
ax=axs[1, 1],
showfliers=False,
palette="gist_rainbow",
)
plt.tight_layout()
plt.show()
Age vs Exited
In [470]:
distribution_plot_wrt_target(Bank_df, 'Age', 'Exited')
Observations
- Distribution of Age is uniform for exiting customers
- Distribution of Age is right skewed for customers who stay with the bank
- There are outliers in both exited/not exited customers
Tenure vs Exited
In [471]:
distribution_plot_wrt_target(Bank_df, 'Tenure', 'Exited')
Observations:
- Tenure, distribution is uniform for both exited/not exited customers
- There are no outliers
- Median is similar for both customer categories
Balance vs Exited
In [472]:
distribution_plot_wrt_target(Bank_df, 'Balance', 'Exited')
Observations:
- Large no. of customers have 0 balance in both exited/not exited categories
- There are no outliers
- Median balance of exited customer is higher than median balance of customers who didnt exit
NumOfProducts vs Exited
In [473]:
distribution_plot_wrt_target(Bank_df, 'NumOfProducts', 'Exited')
Observations:
- Exited customers had used 1 product most with the bank.
- Not exited customers use 2 products most
HasCrCard vs Exited
In [474]:
distribution_plot_wrt_target(Bank_df, 'HasCrCard', 'Exited')
Observations:
- Distribution is similar for both exited/not exited customers
- highest no of customer have 1 creditcard
IsActiveMember vs Exited
In [475]:
distribution_plot_wrt_target(Bank_df, 'IsActiveMember', 'Exited')
Observations:
Exited customers - Highest no. of customers were not active members
Customers who have stayed - Highest no. of customers are active members
EstimatedSalary vs Exited
In [476]:
distribution_plot_wrt_target(Bank_df, 'EstimatedSalary', 'Exited')
Observations:
- Estimated Salary is uniform distribution for both customer categories
- Median salary is the same for both customer categories
- There are no outliers
EDA Summary¶
- There are 10,000 rows and 14 columns.
- There are 11 numerical columns and 3 object column.
- There are no null values.
- There are 79.63% customers who did not leave the bank.
- There are 20.37% customers who left the bank.
- This is an imbalanced dataset.
Univariate Analysis Summary
CreditScore
- CreditScore a left skewed distribution
- Mean and Median are pretty close and similar.
- There are outliers in the data.
Age
- Age is right skewed distribution
- There are outliers
Balance
- There are lot of customers(around 3600) with 0 balance
- distribution is right skewed
NumOfProducts
- there are around 5000 customers using 1 product followed by customers who use 2 products
HasCrCards
- There are around 7000 customers who have credit card and around 3000 customers who dont have credit card.
isActiveMember
- Around 5100 customers are active customers and 4900 customers are inactive customers
EstimatedSalary
- Estimated salary is uniform distribution
Mean and Median are similar.
Exited
- There are around 8000 customer who are with the bank and around 2000 customers who have exited.
Geography
- Customers from France are the highest followed by Germany and Spain
Gender
- Highest number of customers are male.
Bivariate Analysis Summary
From heatmap we observe some postive and negative correlation although none of them are highly correlated.
Positive correlation:
- Age and Exited
Negative Correlation:
- Balance and NumOfProducts
Key relationships with target variable
NumOfProducts vs Exited
- Exited customers had used 1 product most with the bank.
- Not exited customers use 2 products most
isActiveMember
- Exited customers - Highest no. of customers were not active members
- Customers who have stayed - Highest no. of customers are active members
Data Preprocessing¶
In [477]:
Bank_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 12 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Surname 10000 non-null object 1 CreditScore 10000 non-null int64 2 Geography 10000 non-null object 3 Gender 10000 non-null object 4 Age 10000 non-null int64 5 Tenure 10000 non-null int64 6 Balance 10000 non-null float64 7 NumOfProducts 10000 non-null int64 8 HasCrCard 10000 non-null int64 9 IsActiveMember 10000 non-null int64 10 EstimatedSalary 10000 non-null float64 11 Exited 10000 non-null int64 dtypes: float64(2), int64(7), object(3) memory usage: 937.6+ KB
Remove Surname
There are 2932 unique surnames in the dataset.Surname will not add value to the dataset and will not predict customer churn.
In [478]:
#remove Surname
Bank_df.drop(['Surname'], axis=1, inplace=True)
Outlier detection and treatment
In [636]:
# outlier detection using boxplot
numeric_columns = Bank_df.select_dtypes(include=np.number).columns.tolist()
plt.figure(figsize=(15, 12))
for i, variable in enumerate(numeric_columns):
plt.subplot(4, 3, i + 1)
plt.boxplot(Bank_df[variable], whis=1.5)
plt.tight_layout()
plt.title(variable)
plt.show()
Observations:
- Age, CreditScore,NumOfProducts have outliers
- They are all valid and we will not treat the outliers.
Dummy Variable Creation¶
In [479]:
dummy_cols = ['Geography', 'Gender']
Bank_df = pd.get_dummies(Bank_df, columns=dummy_cols, drop_first=True)
Bank_df = Bank_df.astype(float)
In [480]:
Bank_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 12 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 CreditScore 10000 non-null float64 1 Age 10000 non-null float64 2 Tenure 10000 non-null float64 3 Balance 10000 non-null float64 4 NumOfProducts 10000 non-null float64 5 HasCrCard 10000 non-null float64 6 IsActiveMember 10000 non-null float64 7 EstimatedSalary 10000 non-null float64 8 Exited 10000 non-null float64 9 Geography_Germany 10000 non-null float64 10 Geography_Spain 10000 non-null float64 11 Gender_Male 10000 non-null float64 dtypes: float64(12) memory usage: 937.6 KB
In [481]:
X = Bank_df.drop(['Exited'], axis=1)
y = Bank_df['Exited']
Train-validation-test Split¶
In [482]:
# Splitting the dataset into the Training and Test set.
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.2, random_state = 1,stratify = y)
In [483]:
# Splitting the Train dataset into the Training and Validation set.
X_train, X_val, y_train, y_val = train_test_split(X_train,y_train, test_size = 0.2, random_state = 1,stratify = y_train)
In [484]:
#Printing the shapes.
print(X_train.shape,y_train.shape)
print(X_val.shape,y_val.shape)
print(X_test.shape,y_test.shape)
(6400, 11) (6400,) (1600, 11) (1600,) (2000, 11) (2000,)
Data Normalization¶
Columns that we dont need to normalize
- HasCrCard - value 0 or 1. values already in the limited range.
- IsActiveMember - value 0 or 1. Values already in limited range.
- Exited - Target variable and should not be normalized
- Geography_Germany, Geography_Spain, Gender_Male - One-hot encoded features and are already in a normalized range.
In [485]:
normalize_cols = ['CreditScore','Age','Tenure','Balance','NumOfProducts','EstimatedSalary']
scaler = StandardScaler()
scaler.fit(X_train[normalize_cols])
Out[485]:
StandardScaler()In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
StandardScaler()
Once the scaler object fits on the data using the fit() method, it stores the parameters (mean and standard deviation) for normalization based on the training data
We then use these parameters to normalize the validation and test data.
In [486]:
X_train[normalize_cols] = scaler.transform(X_train[normalize_cols])
X_val[normalize_cols] = scaler.transform(X_val[normalize_cols])
X_test[normalize_cols] = scaler.transform(X_test[normalize_cols])
In [487]:
X_train.head()
Out[487]:
| CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Geography_Germany | Geography_Spain | Gender_Male | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 5292 | 0.976155 | -1.047381 | 0.347046 | -1.212523 | 0.796050 | 1.0 | 0.0 | 0.939366 | 0.0 | 0.0 | 0.0 |
| 3879 | 1.316949 | -0.572048 | 1.377238 | 1.214872 | 0.796050 | 0.0 | 0.0 | 1.389376 | 0.0 | 0.0 | 0.0 |
| 6118 | -0.190810 | 0.949016 | 0.003649 | 1.473810 | -0.912441 | 0.0 | 0.0 | 0.692477 | 0.0 | 0.0 | 0.0 |
| 4044 | 0.077695 | -0.667115 | 1.033841 | -1.212523 | -0.912441 | 1.0 | 1.0 | -0.347954 | 0.0 | 0.0 | 1.0 |
| 3202 | -0.893054 | -0.572048 | -0.339749 | 0.961751 | -0.912441 | 1.0 | 0.0 | -0.647963 | 0.0 | 0.0 | 0.0 |
Utility Functions
In [488]:
def plot(history, name):
"""
Function to plot loss/accuracy
history: an object which stores the metrics and losses.
name: can be one of Loss or Accuracy
"""
fig, ax = plt.subplots() #Creating a subplot with figure and axes.
plt.plot(history.history[name]) #Plotting the train accuracy or train loss
plt.plot(history.history['val_'+name]) #Plotting the validation accuracy or validation loss
plt.title('Model ' + name.capitalize()) #Defining the title of the plot.
plt.ylabel(name.capitalize()) #Capitalizing the first letter.
plt.xlabel('Epoch') #Defining the label for the x-axis.
fig.legend(['Train', 'Validation'], loc="outside right upper") #Defining the legend, loc controls the position of the legend.
In [489]:
# defining a function to compute different metrics to check performance of a classification model built using statsmodels
def model_performance_classification(
model, predictors, target, threshold=0.5
):
"""
Function to compute different metrics to check classification model performance
model: classifier
predictors: independent variables
target: dependent variable
threshold: threshold for classifying the observation as class 1
"""
# checking which probabilities are greater than threshold
pred = model.predict(predictors) > threshold
# pred_temp = model.predict(predictors) > threshold
# # rounding off the above values to get classes
# pred = np.round(pred_temp)
acc = accuracy_score(target, pred) # to compute Accuracy
recall = recall_score(target, pred, average='weighted') # to compute Recall
precision = precision_score(target, pred, average='weighted') # to compute Precision
f1 = f1_score(target, pred, average='weighted') # to compute F1-score
# creating a dataframe of metrics
df_perf = pd.DataFrame(
{"Accuracy": acc, "Recall": recall, "Precision": precision, "F1 Score": f1,},
index=[0],
)
return df_perf
Model Building¶
Model Evaluation Criterion¶
Write down the logic for choosing the metric that would be the best metric for this business scenario.
Model can make wrong predictions as:
Model predicts that the customer will exit the bank,but the customer doesnt exit the bank - FP
Models predicts that the customer will not exit the bank, but the customer exit the bank - FN
Which case is more important? Failing to predict the customer will exit the bank could result in loosing the customer(FN)
How to reduce this loss?
Maximize Recall to reduce False Negatives.
In [490]:
X_train.info()
<class 'pandas.core.frame.DataFrame'> Index: 6400 entries, 5292 to 4847 Data columns (total 11 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 CreditScore 6400 non-null float64 1 Age 6400 non-null float64 2 Tenure 6400 non-null float64 3 Balance 6400 non-null float64 4 NumOfProducts 6400 non-null float64 5 HasCrCard 6400 non-null float64 6 IsActiveMember 6400 non-null float64 7 EstimatedSalary 6400 non-null float64 8 Geography_Germany 6400 non-null float64 9 Geography_Spain 6400 non-null float64 10 Gender_Male 6400 non-null float64 dtypes: float64(11) memory usage: 600.0 KB
In [491]:
# defining the batch size and # epochs upfront as we'll be using the same values for all models
epochs = 25
batch_size = 64
Model 0 - Neural Network with SGD Optimizer(without class weights)¶
- Let's start with a neural network consisting of
- two hidden layers with 14 and 7 neurons respectively
- activation function of ReLU.
- SGD as the optimizer
In [492]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [493]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dense(1,activation="sigmoid"))
In [494]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [495]:
optimizer = tf.keras.optimizers.SGD() # defining SGD as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [496]:
start = time.time()
history = model.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.8444 - val_loss: 0.6672 Epoch 2/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6426 - val_loss: 0.5911 Epoch 3/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5805 - val_loss: 0.5559 Epoch 4/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5504 - val_loss: 0.5366 Epoch 5/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5335 - val_loss: 0.5250 Epoch 6/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5233 - val_loss: 0.5177 Epoch 7/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5168 - val_loss: 0.5129 Epoch 8/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5125 - val_loss: 0.5096 Epoch 9/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5095 - val_loss: 0.5073 Epoch 10/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5073 - val_loss: 0.5055 Epoch 11/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5057 - val_loss: 0.5041 Epoch 12/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5043 - val_loss: 0.5029 Epoch 13/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5032 - val_loss: 0.5019 Epoch 14/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5021 - val_loss: 0.5008 Epoch 15/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5011 - val_loss: 0.4998 Epoch 16/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5000 - val_loss: 0.4988 Epoch 17/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4989 - val_loss: 0.4976 Epoch 18/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4976 - val_loss: 0.4964 Epoch 19/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4962 - val_loss: 0.4949 Epoch 20/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4946 - val_loss: 0.4933 Epoch 21/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4928 - val_loss: 0.4914 Epoch 22/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4908 - val_loss: 0.4894 Epoch 23/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4885 - val_loss: 0.4871 Epoch 24/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4859 - val_loss: 0.4845 Epoch 25/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4829 - val_loss: 0.4816
In [497]:
print("Time taken in seconds ",end-start)
Time taken in seconds 9.274894714355469
In [498]:
plot(history,'loss')
In [499]:
model_0_train_perf = model_performance_classification(model, X_train, y_train)
model_0_train_perf
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[499]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.79625 | 0.79625 | 0.634014 | 0.705931 |
In [500]:
model_0_val_perf = model_performance_classification(model, X_val, y_val)
model_0_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[500]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.79625 | 0.79625 | 0.634014 | 0.705931 |
Observations:
- Recall score is .79625 in training and 0.79625 with validation dataset.
Model 1 - Neural Network with SGD Optimizer(class weights)¶
Using class weights even though the dataset is mild imbalance
In [501]:
# Calculate class weights for imbalanced dataset
cw = (y_train.shape[0]) / np.bincount(y_train)
# Create a dictionary mapping class indices to their respective class weights
cw_dict = {}
for i in range(cw.shape[0]):
cw_dict[i] = cw[i]
cw_dict
Out[501]:
{0: 1.2558869701726845, 1: 4.9079754601226995}
In [502]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [503]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dense(1,activation="sigmoid"))
In [504]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [505]:
optimizer = tf.keras.optimizers.SGD() # defining SGD as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [506]:
start = time.time()
history = model.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs,class_weight=cw_dict)
end=time.time()
Epoch 1/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 7ms/step - loss: 1.5033 - val_loss: 0.6950 Epoch 2/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 1.3686 - val_loss: 0.6606 Epoch 3/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.3159 - val_loss: 0.6350 Epoch 4/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.2756 - val_loss: 0.6129 Epoch 5/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.2417 - val_loss: 0.5953 Epoch 6/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.2144 - val_loss: 0.5810 Epoch 7/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.1912 - val_loss: 0.5686 Epoch 8/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.1702 - val_loss: 0.5570 Epoch 9/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.1499 - val_loss: 0.5464 Epoch 10/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.1294 - val_loss: 0.5358 Epoch 11/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.1093 - val_loss: 0.5264 Epoch 12/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.0900 - val_loss: 0.5176 Epoch 13/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 1.0723 - val_loss: 0.5090 Epoch 14/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 1.0555 - val_loss: 0.5011 Epoch 15/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.0400 - val_loss: 0.4942 Epoch 16/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.0258 - val_loss: 0.4885 Epoch 17/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.0132 - val_loss: 0.4841 Epoch 18/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 1.0023 - val_loss: 0.4801 Epoch 19/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.9929 - val_loss: 0.4763 Epoch 20/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9847 - val_loss: 0.4731 Epoch 21/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9778 - val_loss: 0.4702 Epoch 22/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9719 - val_loss: 0.4679 Epoch 23/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.9668 - val_loss: 0.4656 Epoch 24/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.9622 - val_loss: 0.4637 Epoch 25/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9581 - val_loss: 0.4619
In [507]:
print("Time taken in seconds ",end-start)
Time taken in seconds 10.794859409332275
In [508]:
plot(history,'loss')
In [509]:
model_1_train_perf = model_performance_classification(model, X_train, y_train)
model_1_train_perf
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[509]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.7775 | 0.7775 | 0.832367 | 0.793818 |
In [510]:
model_1_val_perf = model_performance_classification(model, X_val, y_val)
model_1_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[510]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.78125 | 0.78125 | 0.833338 | 0.796914 |
Observations:
- Recall for training is 0.7775 and 0.78125 for validation dataset.
- Model without adding class weights performed better.
Add momentum to the model with class weights
Model 2 - Neural Network with SGD Optimizer(class weights) with Momentum¶
In [511]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [512]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dense(1,activation="sigmoid"))
In [513]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [514]:
optimizer = tf.keras.optimizers.SGD(momentum=0.9) # defining SGD as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [515]:
start = time.time()
history = model.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs,class_weight = cw_dict)
end=time.time()
Epoch 1/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 2s 7ms/step - loss: 1.2809 - val_loss: 0.4968 Epoch 2/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 1.0510 - val_loss: 0.4650 Epoch 3/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9729 - val_loss: 0.4542 Epoch 4/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9509 - val_loss: 0.4452 Epoch 5/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9383 - val_loss: 0.4407 Epoch 6/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9310 - val_loss: 0.4364 Epoch 7/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.9255 - val_loss: 0.4352 Epoch 8/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9212 - val_loss: 0.4337 Epoch 9/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.9203 - val_loss: 0.4317 Epoch 10/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.9176 - val_loss: 0.4301 Epoch 11/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9156 - val_loss: 0.4289 Epoch 12/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.9136 - val_loss: 0.4279 Epoch 13/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9116 - val_loss: 0.4293 Epoch 14/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9102 - val_loss: 0.4272 Epoch 15/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9080 - val_loss: 0.4270 Epoch 16/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9063 - val_loss: 0.4278 Epoch 17/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9040 - val_loss: 0.4272 Epoch 18/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.9029 - val_loss: 0.4263 Epoch 19/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9019 - val_loss: 0.4268 Epoch 20/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.9011 - val_loss: 0.4265 Epoch 21/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.8998 - val_loss: 0.4283 Epoch 22/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.8991 - val_loss: 0.4275 Epoch 23/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.8977 - val_loss: 0.4277 Epoch 24/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.8963 - val_loss: 0.4277 Epoch 25/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.8957 - val_loss: 0.4280
In [516]:
print("Time taken in seconds ",end-start)
Time taken in seconds 11.847717523574829
In [517]:
plot(history,'loss')
In [518]:
model_2_train_perf = model_performance_classification(model, X_train, y_train)
model_2_train_perf
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[518]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.812031 | 0.812031 | 0.845554 | 0.822852 |
In [519]:
model_2_val_perf = model_performance_classification(model, X_val, y_val)
model_2_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[519]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.803125 | 0.803125 | 0.833083 | 0.813496 |
Observations:
- After adding momentum, we see the Model has generalised better when compared to the other 2 models.
- Recall is 0.812031 for training dataset and 0.803125 for validation dataset
Lets do model performance improvement
Model Performance Improvement¶
Build a model using Adam optimizer
Model 3 - Neural Network with Adam Optimizer¶
In [520]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [521]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dense(1,activation="sigmoid"))
In [522]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [523]:
optimizer = tf.keras.optimizers.Adam() # defining Adam as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [524]:
start = time.time()
history = model.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - loss: 0.7392 - val_loss: 0.5363 Epoch 2/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5096 - val_loss: 0.4596 Epoch 3/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4531 - val_loss: 0.4316 Epoch 4/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4331 - val_loss: 0.4190 Epoch 5/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4201 - val_loss: 0.4069 Epoch 6/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4041 - val_loss: 0.3926 Epoch 7/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3868 - val_loss: 0.3813 Epoch 8/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.3730 - val_loss: 0.3733 Epoch 9/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3631 - val_loss: 0.3677 Epoch 10/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3562 - val_loss: 0.3645 Epoch 11/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3514 - val_loss: 0.3624 Epoch 12/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3483 - val_loss: 0.3608 Epoch 13/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3459 - val_loss: 0.3599 Epoch 14/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3441 - val_loss: 0.3591 Epoch 15/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3426 - val_loss: 0.3585 Epoch 16/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3414 - val_loss: 0.3581 Epoch 17/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3404 - val_loss: 0.3578 Epoch 18/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3394 - val_loss: 0.3576 Epoch 19/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3386 - val_loss: 0.3575 Epoch 20/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3378 - val_loss: 0.3572 Epoch 21/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3371 - val_loss: 0.3569 Epoch 22/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.3364 - val_loss: 0.3568 Epoch 23/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.3358 - val_loss: 0.3565 Epoch 24/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.3353 - val_loss: 0.3563 Epoch 25/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.3348 - val_loss: 0.3563
In [525]:
print("Time taken in seconds ",end-start)
Time taken in seconds 13.720810174942017
In [526]:
plot(history,'loss')
In [527]:
model_3_train_perf = model_performance_classification(model, X_train, y_train)
model_3_train_perf
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[527]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.859688 | 0.859688 | 0.851531 | 0.843477 |
In [528]:
model_3_val_perf = model_performance_classification(model, X_val, y_val)
model_3_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[528]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.8475 | 0.8475 | 0.836086 | 0.827564 |
Observations:
- Recall for training is 0.859668 and 0.8475 for validation dataset.
- We see that the error continues to decrease as epoch increases.
- So next step is to build the model with different Epoch and Batch size.
Model 4 - Neural Network with Adam Optimizer(Change Epoch and Batch Size)¶
In [529]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [530]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dense(1,activation="sigmoid"))
In [531]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [532]:
optimizer = tf.keras.optimizers.Adam() # defining Adam as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [533]:
#choose new epoch and batch size
new_epochs = 50
new_batch_size = 100
In [534]:
start = time.time()
history = model.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=new_batch_size, epochs=new_epochs)
end=time.time()
Epoch 1/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - loss: 0.6755 - val_loss: 0.5126 Epoch 2/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4938 - val_loss: 0.4670 Epoch 3/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4589 - val_loss: 0.4507 Epoch 4/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4446 - val_loss: 0.4407 Epoch 5/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4358 - val_loss: 0.4338 Epoch 6/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4289 - val_loss: 0.4275 Epoch 7/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4223 - val_loss: 0.4211 Epoch 8/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4151 - val_loss: 0.4137 Epoch 9/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4073 - val_loss: 0.4058 Epoch 10/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3992 - val_loss: 0.3978 Epoch 11/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3910 - val_loss: 0.3900 Epoch 12/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3834 - val_loss: 0.3837 Epoch 13/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3768 - val_loss: 0.3786 Epoch 14/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3714 - val_loss: 0.3748 Epoch 15/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3672 - val_loss: 0.3720 Epoch 16/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3638 - val_loss: 0.3700 Epoch 17/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3612 - val_loss: 0.3684 Epoch 18/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3591 - val_loss: 0.3671 Epoch 19/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3573 - val_loss: 0.3662 Epoch 20/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3559 - val_loss: 0.3654 Epoch 21/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3547 - val_loss: 0.3649 Epoch 22/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3537 - val_loss: 0.3646 Epoch 23/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3527 - val_loss: 0.3643 Epoch 24/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3519 - val_loss: 0.3639 Epoch 25/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3512 - val_loss: 0.3636 Epoch 26/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3505 - val_loss: 0.3634 Epoch 27/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step - loss: 0.3499 - val_loss: 0.3631 Epoch 28/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.3494 - val_loss: 0.3629 Epoch 29/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.3488 - val_loss: 0.3626 Epoch 30/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - loss: 0.3483 - val_loss: 0.3624 Epoch 31/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.3477 - val_loss: 0.3622 Epoch 32/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3472 - val_loss: 0.3620 Epoch 33/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3467 - val_loss: 0.3619 Epoch 34/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3462 - val_loss: 0.3619 Epoch 35/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3457 - val_loss: 0.3618 Epoch 36/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3452 - val_loss: 0.3617 Epoch 37/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3446 - val_loss: 0.3617 Epoch 38/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3440 - val_loss: 0.3618 Epoch 39/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3434 - val_loss: 0.3618 Epoch 40/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3429 - val_loss: 0.3617 Epoch 41/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3423 - val_loss: 0.3617 Epoch 42/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3418 - val_loss: 0.3617 Epoch 43/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3413 - val_loss: 0.3615 Epoch 44/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3408 - val_loss: 0.3615 Epoch 45/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3403 - val_loss: 0.3615 Epoch 46/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3399 - val_loss: 0.3615 Epoch 47/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3393 - val_loss: 0.3614 Epoch 48/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3388 - val_loss: 0.3613 Epoch 49/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3383 - val_loss: 0.3612 Epoch 50/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3378 - val_loss: 0.3610
In [535]:
print("Time taken in seconds ",end-start)
Time taken in seconds 16.66175889968872
In [536]:
plot(history,'loss')
In [537]:
model_4_train_perf = model_performance_classification(model, X_train, y_train)
model_4_train_perf
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[537]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.859062 | 0.859062 | 0.849094 | 0.846678 |
In [538]:
model_4_val_perf = model_performance_classification(model, X_val, y_val)
model_4_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[538]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.841875 | 0.841875 | 0.827511 | 0.826503 |
Observations:
- Model has not performed better with change in Epoch and Batch size
- Recall is 0.859062 for training set and 0.841875 for validation dataset.
Model 5 - Neural Network with Adam Optimizer and Dropout¶
-Lets build a model with Adam Optimizer and dropout for both hidden layer(0.4,0.2)
In [539]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [540]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train.shape[1]))
model.add(Dropout(0.4)) #add drop out
model.add(Dense(7,activation="relu"))
model.add(Dropout(0.2)) # add drop out
model.add(Dense(1,activation="sigmoid"))
In [541]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dropout (Dropout) │ (None, 14) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dropout_1 (Dropout) │ (None, 7) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [542]:
optimizer = tf.keras.optimizers.Adam() # defining Adam as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [543]:
start = time.time()
history = model.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - loss: 0.7430 - val_loss: 0.6227 Epoch 2/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6018 - val_loss: 0.4986 Epoch 3/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5245 - val_loss: 0.4565 Epoch 4/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4914 - val_loss: 0.4371 Epoch 5/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4852 - val_loss: 0.4250 Epoch 6/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4683 - val_loss: 0.4159 Epoch 7/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4567 - val_loss: 0.4074 Epoch 8/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4471 - val_loss: 0.3993 Epoch 9/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4379 - val_loss: 0.3933 Epoch 10/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4327 - val_loss: 0.3893 Epoch 11/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4334 - val_loss: 0.3862 Epoch 12/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4298 - val_loss: 0.3832 Epoch 13/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4110 - val_loss: 0.3794 Epoch 14/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4128 - val_loss: 0.3778 Epoch 15/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4085 - val_loss: 0.3771 Epoch 16/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4134 - val_loss: 0.3759 Epoch 17/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4135 - val_loss: 0.3753 Epoch 18/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4101 - val_loss: 0.3725 Epoch 19/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4126 - val_loss: 0.3723 Epoch 20/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4109 - val_loss: 0.3721 Epoch 21/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4007 - val_loss: 0.3695 Epoch 22/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4039 - val_loss: 0.3689 Epoch 23/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4025 - val_loss: 0.3662 Epoch 24/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3856 - val_loss: 0.3632 Epoch 25/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3952 - val_loss: 0.3621
In [544]:
print("Time taken in seconds ",end-start)
Time taken in seconds 14.420158624649048
In [545]:
plot(history,'loss')
In [546]:
model_5_train_perf = model_performance_classification(model, X_train, y_train)
model_5_train_perf
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[546]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.855313 | 0.855313 | 0.847971 | 0.835461 |
In [547]:
model_5_val_perf = model_performance_classification(model, X_val, y_val)
model_5_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[547]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.84375 | 0.84375 | 0.832212 | 0.820695 |
Observations:
- With dropouts in hidden layer, there is no much change in the model performance.
- Recall is 0.855313 for training and 0.84375 for validation dataset.
Model 6 - Neural Network with Adam Optimizer and Dropout(Only in one hidden layer)¶
In [548]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [549]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dropout(0.2))
model.add(Dense(1,activation="sigmoid"))
In [550]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dropout (Dropout) │ (None, 7) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [551]:
optimizer = tf.keras.optimizers.Adam() # defining Adam as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [552]:
start = time.time()
history = model.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 2s 6ms/step - loss: 0.6932 - val_loss: 0.5447 Epoch 2/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.5411 - val_loss: 0.4891 Epoch 3/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4926 - val_loss: 0.4599 Epoch 4/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4649 - val_loss: 0.4448 Epoch 5/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4590 - val_loss: 0.4360 Epoch 6/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4523 - val_loss: 0.4292 Epoch 7/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4360 - val_loss: 0.4226 Epoch 8/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4338 - val_loss: 0.4164 Epoch 9/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4259 - val_loss: 0.4105 Epoch 10/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4215 - val_loss: 0.4055 Epoch 11/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4113 - val_loss: 0.4007 Epoch 12/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4152 - val_loss: 0.3967 Epoch 13/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4046 - val_loss: 0.3921 Epoch 14/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4026 - val_loss: 0.3881 Epoch 15/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3928 - val_loss: 0.3842 Epoch 16/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3930 - val_loss: 0.3813 Epoch 17/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3869 - val_loss: 0.3786 Epoch 18/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3885 - val_loss: 0.3761 Epoch 19/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3807 - val_loss: 0.3731 Epoch 20/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3784 - val_loss: 0.3711 Epoch 21/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3716 - val_loss: 0.3696 Epoch 22/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3721 - val_loss: 0.3667 Epoch 23/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3699 - val_loss: 0.3657 Epoch 24/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3770 - val_loss: 0.3643 Epoch 25/25 100/100 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.3567 - val_loss: 0.3624
In [553]:
print("Time taken in seconds ",end-start)
Time taken in seconds 12.597172021865845
In [554]:
plot(history,'loss')
In [555]:
model_6_train_perf = model_performance_classification(model, X_train, y_train)
model_6_train_perf
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[555]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.855938 | 0.855938 | 0.849774 | 0.835303 |
In [556]:
model_6_val_perf = model_performance_classification(model, X_val, y_val)
model_6_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[556]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.844375 | 0.844375 | 0.833886 | 0.820434 |
Observations:
- Model with 1 dropout is better than model with 2 dropouts
- Recall is 0.855938 for training and 0.844375 for validation dataset.
- Choose the model with 1 drop out and change the Epoch/Batch size
Model 7 - Neural Network with Adam Optimizer, Dropout, Change Epochs and Batch Size¶
In [557]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [558]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dropout(0.2))
model.add(Dense(1,activation="sigmoid"))
In [559]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dropout (Dropout) │ (None, 7) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [560]:
optimizer = tf.keras.optimizers.Adam() # defining Adam as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [561]:
start = time.time()
history = model.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=new_batch_size, epochs=new_epochs)
end=time.time()
Epoch 1/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 2s 6ms/step - loss: 0.7084 - val_loss: 0.6315 Epoch 2/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.6045 - val_loss: 0.5308 Epoch 3/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5220 - val_loss: 0.4807 Epoch 4/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4940 - val_loss: 0.4591 Epoch 5/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4704 - val_loss: 0.4410 Epoch 6/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4599 - val_loss: 0.4257 Epoch 7/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4394 - val_loss: 0.4147 Epoch 8/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4296 - val_loss: 0.4043 Epoch 9/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4141 - val_loss: 0.3953 Epoch 10/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4088 - val_loss: 0.3877 Epoch 11/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3962 - val_loss: 0.3816 Epoch 12/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3971 - val_loss: 0.3767 Epoch 13/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3904 - val_loss: 0.3721 Epoch 14/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3837 - val_loss: 0.3687 Epoch 15/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3787 - val_loss: 0.3659 Epoch 16/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3726 - val_loss: 0.3633 Epoch 17/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3698 - val_loss: 0.3613 Epoch 18/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3633 - val_loss: 0.3604 Epoch 19/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3664 - val_loss: 0.3589 Epoch 20/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3712 - val_loss: 0.3587 Epoch 21/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3697 - val_loss: 0.3581 Epoch 22/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3644 - val_loss: 0.3570 Epoch 23/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3623 - val_loss: 0.3561 Epoch 24/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3648 - val_loss: 0.3557 Epoch 25/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3674 - val_loss: 0.3549 Epoch 26/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3602 - val_loss: 0.3543 Epoch 27/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3650 - val_loss: 0.3544 Epoch 28/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3563 - val_loss: 0.3541 Epoch 29/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3616 - val_loss: 0.3541 Epoch 30/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3615 - val_loss: 0.3538 Epoch 31/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3617 - val_loss: 0.3532 Epoch 32/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3561 - val_loss: 0.3527 Epoch 33/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3600 - val_loss: 0.3519 Epoch 34/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3602 - val_loss: 0.3520 Epoch 35/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.3543 - val_loss: 0.3518 Epoch 36/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - loss: 0.3529 - val_loss: 0.3522 Epoch 37/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - loss: 0.3543 - val_loss: 0.3520 Epoch 38/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 7ms/step - loss: 0.3523 - val_loss: 0.3518 Epoch 39/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3510 - val_loss: 0.3519 Epoch 40/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3551 - val_loss: 0.3515 Epoch 41/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3571 - val_loss: 0.3514 Epoch 42/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3539 - val_loss: 0.3512 Epoch 43/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3528 - val_loss: 0.3511 Epoch 44/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3563 - val_loss: 0.3512 Epoch 45/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3503 - val_loss: 0.3506 Epoch 46/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3554 - val_loss: 0.3505 Epoch 47/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3517 - val_loss: 0.3504 Epoch 48/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3514 - val_loss: 0.3509 Epoch 49/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3490 - val_loss: 0.3510 Epoch 50/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3532 - val_loss: 0.3508
In [562]:
print("Time taken in seconds ",end-start)
Time taken in seconds 17.837464570999146
In [563]:
plot(history,'loss')
In [564]:
model_7_train_perf = model_performance_classification(model, X_train, y_train)
model_7_train_perf
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[564]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.863906 | 0.863906 | 0.855526 | 0.850778 |
In [565]:
model_7_val_perf = model_performance_classification(model, X_val, y_val)
model_7_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[565]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.85 | 0.85 | 0.83795 | 0.834656 |
Observations:
- Model with dropout on one hidden layer with increased epoc and batch size has generalised better than dropouts with 2 hidden layers and with less epocs/batch size.
- Recall is 0.863906 for training and 0.85 for validation dataset.
Model 8 - Neural Network with Balanced Data (by applying SMOTE) and SGD Optimizer¶
In [566]:
# Fit SMOTE on train data(Synthetic Minority Oversampling Technique)
sm = SMOTE(sampling_strategy=1, k_neighbors=5, random_state=1)
X_train_over, y_train_over = sm.fit_resample(X_train, y_train)
In [567]:
print("Before OverSampling, count of label '1': {}".format(sum(y_train == 1)))
print("Before OverSampling, count of label '0': {} \n".format(sum(y_train == 0)))
print("After OverSampling, count of label '1': {}".format(sum(y_train_over == 1)))
print("After OverSampling, count of label '0': {} \n".format(sum(y_train_over == 0)))
print("After OverSampling, the shape of train_X: {}".format(X_train_over.shape))
print("After OverSampling, the shape of train_y: {} \n".format(y_train_over.shape))
Before OverSampling, count of label '1': 1304 Before OverSampling, count of label '0': 5096 After OverSampling, count of label '1': 5096 After OverSampling, count of label '0': 5096 After OverSampling, the shape of train_X: (10192, 11) After OverSampling, the shape of train_y: (10192,)
In [568]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [569]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train_over.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dense(1,activation="sigmoid"))
In [570]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [571]:
optimizer = tf.keras.optimizers.SGD() # defining SGD as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [572]:
start = time.time()
history = model.fit(X_train_over, y_train_over, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.7135 - val_loss: 0.6922 Epoch 2/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6679 - val_loss: 0.6625 Epoch 3/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6461 - val_loss: 0.6495 Epoch 4/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.6315 - val_loss: 0.6389 Epoch 5/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6200 - val_loss: 0.6279 Epoch 6/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6102 - val_loss: 0.6180 Epoch 7/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6016 - val_loss: 0.6081 Epoch 8/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5936 - val_loss: 0.5992 Epoch 9/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.5858 - val_loss: 0.5896 Epoch 10/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.5773 - val_loss: 0.5788 Epoch 11/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.5687 - val_loss: 0.5675 Epoch 12/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.5600 - val_loss: 0.5581 Epoch 13/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.5517 - val_loss: 0.5506 Epoch 14/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5440 - val_loss: 0.5427 Epoch 15/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5369 - val_loss: 0.5368 Epoch 16/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5302 - val_loss: 0.5314 Epoch 17/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5237 - val_loss: 0.5261 Epoch 18/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5175 - val_loss: 0.5205 Epoch 19/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5115 - val_loss: 0.5157 Epoch 20/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5058 - val_loss: 0.5105 Epoch 21/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5003 - val_loss: 0.5061 Epoch 22/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4951 - val_loss: 0.5017 Epoch 23/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4901 - val_loss: 0.4979 Epoch 24/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4854 - val_loss: 0.4945 Epoch 25/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4809 - val_loss: 0.4910
In [573]:
print("Time taken in seconds ",end-start)
Time taken in seconds 13.167799234390259
In [574]:
plot(history,'loss')
In [575]:
model_8_train_perf = model_performance_classification(model, X_train_over, y_train_over)
model_8_train_perf
319/319 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
Out[575]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.769231 | 0.769231 | 0.769428 | 0.769188 |
In [576]:
model_8_val_perf = model_performance_classification(model, X_val, y_val)
model_8_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[576]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.7675 | 0.7675 | 0.834171 | 0.786109 |
Observations:
- Recall for training is 0.769231 and 0.7675 for validation dataset.
- Model hasnt performed well when compared to other models we have seen so far.
Model 9 - Neural Network with Balanced Data (by applying SMOTE),SGD Optimizer,Change Epoch and Batch Size¶
Lets increase Epoch and Batch size
In [577]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [578]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train_over.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dense(1,activation="sigmoid"))
In [579]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [580]:
optimizer = tf.keras.optimizers.SGD() # defining SGD as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [581]:
start = time.time()
history = model.fit(X_train_over, y_train_over, validation_data=(X_val,y_val) , batch_size=new_batch_size, epochs=new_epochs)
end=time.time()
Epoch 1/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.7575 - val_loss: 0.6570 Epoch 2/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6945 - val_loss: 0.6733 Epoch 3/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6774 - val_loss: 0.6765 Epoch 4/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.6673 - val_loss: 0.6730 Epoch 5/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6585 - val_loss: 0.6659 Epoch 6/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step - loss: 0.6491 - val_loss: 0.6565 Epoch 7/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.6388 - val_loss: 0.6453 Epoch 8/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.6280 - val_loss: 0.6334 Epoch 9/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.6169 - val_loss: 0.6210 Epoch 10/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.6055 - val_loss: 0.6087 Epoch 11/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.5944 - val_loss: 0.5963 Epoch 12/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5836 - val_loss: 0.5842 Epoch 13/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5735 - val_loss: 0.5727 Epoch 14/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5639 - val_loss: 0.5615 Epoch 15/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5549 - val_loss: 0.5510 Epoch 16/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5464 - val_loss: 0.5416 Epoch 17/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5385 - val_loss: 0.5331 Epoch 18/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5310 - val_loss: 0.5255 Epoch 19/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5238 - val_loss: 0.5189 Epoch 20/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5169 - val_loss: 0.5126 Epoch 21/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5105 - val_loss: 0.5070 Epoch 22/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5045 - val_loss: 0.5018 Epoch 23/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4990 - val_loss: 0.4972 Epoch 24/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4940 - val_loss: 0.4933 Epoch 25/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4893 - val_loss: 0.4895 Epoch 26/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4850 - val_loss: 0.4858 Epoch 27/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4812 - val_loss: 0.4829 Epoch 28/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4776 - val_loss: 0.4799 Epoch 29/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4744 - val_loss: 0.4774 Epoch 30/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4716 - val_loss: 0.4751 Epoch 31/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4689 - val_loss: 0.4728 Epoch 32/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4665 - val_loss: 0.4706 Epoch 33/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4643 - val_loss: 0.4686 Epoch 34/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4624 - val_loss: 0.4667 Epoch 35/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4606 - val_loss: 0.4652 Epoch 36/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4589 - val_loss: 0.4637 Epoch 37/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4573 - val_loss: 0.4624 Epoch 38/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4559 - val_loss: 0.4613 Epoch 39/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4545 - val_loss: 0.4603 Epoch 40/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4532 - val_loss: 0.4593 Epoch 41/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4521 - val_loss: 0.4583 Epoch 42/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4510 - val_loss: 0.4575 Epoch 43/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4501 - val_loss: 0.4567 Epoch 44/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4491 - val_loss: 0.4560 Epoch 45/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4483 - val_loss: 0.4553 Epoch 46/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4475 - val_loss: 0.4547 Epoch 47/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4467 - val_loss: 0.4542 Epoch 48/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4460 - val_loss: 0.4537 Epoch 49/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4453 - val_loss: 0.4532 Epoch 50/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4447 - val_loss: 0.4527
In [582]:
print("Time taken in seconds ",end-start)
Time taken in seconds 21.994659900665283
In [583]:
plot(history,'loss')
In [584]:
model_9_train_perf = model_performance_classification(model, X_train_over, y_train_over)
model_9_train_perf
319/319 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
Out[584]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.783458 | 0.783458 | 0.783461 | 0.783457 |
In [585]:
model_9_val_perf = model_performance_classification(model, X_val, y_val)
model_9_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[585]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.78125 | 0.78125 | 0.832699 | 0.79681 |
Observations:
- Recall is 0.783458 for training and 0.78125 for validation dataset
- After increase in epoch,batchsize,its slightly better although the overal performance of the model is lower when compared to other models.
Model 10 - Neural Network with Balanced Data (by applying SMOTE) and Adam Optimizer¶
In [586]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [587]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train_over.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dense(1,activation="sigmoid"))
In [588]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [589]:
optimizer = tf.keras.optimizers.Adam() # defining Adam as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [590]:
start = time.time()
history = model.fit(X_train_over, y_train_over, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - loss: 0.6640 - val_loss: 0.5826 Epoch 2/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5487 - val_loss: 0.5070 Epoch 3/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4820 - val_loss: 0.4846 Epoch 4/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4582 - val_loss: 0.4747 Epoch 5/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4481 - val_loss: 0.4678 Epoch 6/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4421 - val_loss: 0.4650 Epoch 7/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4379 - val_loss: 0.4631 Epoch 8/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4347 - val_loss: 0.4609 Epoch 9/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4319 - val_loss: 0.4594 Epoch 10/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4295 - val_loss: 0.4585 Epoch 11/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4273 - val_loss: 0.4576 Epoch 12/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4251 - val_loss: 0.4570 Epoch 13/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4233 - val_loss: 0.4561 Epoch 14/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4217 - val_loss: 0.4557 Epoch 15/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4203 - val_loss: 0.4554 Epoch 16/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4189 - val_loss: 0.4549 Epoch 17/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4176 - val_loss: 0.4545 Epoch 18/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4164 - val_loss: 0.4536 Epoch 19/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4153 - val_loss: 0.4529 Epoch 20/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4142 - val_loss: 0.4531 Epoch 21/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4133 - val_loss: 0.4532 Epoch 22/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4124 - val_loss: 0.4536 Epoch 23/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4115 - val_loss: 0.4537 Epoch 24/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4106 - val_loss: 0.4543 Epoch 25/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4098 - val_loss: 0.4544
In [591]:
print("Time taken in seconds ",end-start)
Time taken in seconds 16.479286193847656
In [592]:
plot(history,'loss')
In [593]:
model_10_train_perf = model_performance_classification(model, X_train_over, y_train_over)
model_10_train_perf
319/319 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
Out[593]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.806711 | 0.806711 | 0.806722 | 0.806709 |
In [594]:
model_10_val_perf = model_performance_classification(model, X_val, y_val)
model_10_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[594]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.783125 | 0.783125 | 0.834075 | 0.7985 |
Observations:
- With SMOTE, and Adam Optimizer, the model has a recall score for 0.806711 for training dataset and 0.783125 for validation data set.
Model 11 - Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Change Epoch and Batch size¶
Lets increase the Epoch and Batch size
In [595]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [596]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train_over.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dense(1,activation="sigmoid"))
In [597]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [598]:
optimizer = tf.keras.optimizers.Adam() # defining Adam as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [599]:
start = time.time()
history = model.fit(X_train_over, y_train_over, validation_data=(X_val,y_val) , batch_size=new_batch_size, epochs=new_epochs)
end=time.time()
Epoch 1/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 2s 8ms/step - loss: 0.6808 - val_loss: 0.6191 Epoch 2/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.6003 - val_loss: 0.5546 Epoch 3/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5423 - val_loss: 0.5180 Epoch 4/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5048 - val_loss: 0.4968 Epoch 5/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4811 - val_loss: 0.4829 Epoch 6/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4668 - val_loss: 0.4735 Epoch 7/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4580 - val_loss: 0.4683 Epoch 8/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4523 - val_loss: 0.4636 Epoch 9/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4481 - val_loss: 0.4617 Epoch 10/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4446 - val_loss: 0.4592 Epoch 11/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4415 - val_loss: 0.4572 Epoch 12/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4387 - val_loss: 0.4558 Epoch 13/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4361 - val_loss: 0.4547 Epoch 14/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4337 - val_loss: 0.4546 Epoch 15/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4317 - val_loss: 0.4543 Epoch 16/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4298 - val_loss: 0.4538 Epoch 17/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4281 - val_loss: 0.4537 Epoch 18/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4265 - val_loss: 0.4545 Epoch 19/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4249 - val_loss: 0.4549 Epoch 20/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4235 - val_loss: 0.4547 Epoch 21/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4222 - val_loss: 0.4550 Epoch 22/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4211 - val_loss: 0.4550 Epoch 23/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4202 - val_loss: 0.4550 Epoch 24/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4192 - val_loss: 0.4545 Epoch 25/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4182 - val_loss: 0.4546 Epoch 26/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4173 - val_loss: 0.4544 Epoch 27/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4163 - val_loss: 0.4543 Epoch 28/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4156 - val_loss: 0.4543 Epoch 29/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4148 - val_loss: 0.4547 Epoch 30/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4141 - val_loss: 0.4540 Epoch 31/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4135 - val_loss: 0.4542 Epoch 32/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4128 - val_loss: 0.4543 Epoch 33/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4121 - val_loss: 0.4540 Epoch 34/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4115 - val_loss: 0.4537 Epoch 35/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4109 - val_loss: 0.4540 Epoch 36/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4103 - val_loss: 0.4540 Epoch 37/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4097 - val_loss: 0.4542 Epoch 38/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4091 - val_loss: 0.4541 Epoch 39/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4085 - val_loss: 0.4541 Epoch 40/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4079 - val_loss: 0.4542 Epoch 41/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4074 - val_loss: 0.4542 Epoch 42/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4069 - val_loss: 0.4543 Epoch 43/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4065 - val_loss: 0.4540 Epoch 44/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4060 - val_loss: 0.4540 Epoch 45/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4056 - val_loss: 0.4538 Epoch 46/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4052 - val_loss: 0.4538 Epoch 47/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4048 - val_loss: 0.4537 Epoch 48/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4044 - val_loss: 0.4536 Epoch 49/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4040 - val_loss: 0.4537 Epoch 50/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4037 - val_loss: 0.4537
In [600]:
print("Time taken in seconds ",end-start)
Time taken in seconds 27.989372491836548
In [601]:
plot(history,'loss')
In [602]:
model_11_train_perf = model_performance_classification(model, X_train_over, y_train_over)
model_11_train_perf
319/319 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step
Out[602]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.807987 | 0.807987 | 0.808011 | 0.807983 |
In [603]:
model_11_val_perf = model_performance_classification(model, X_val, y_val)
model_11_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[603]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.796875 | 0.796875 | 0.836055 | 0.809459 |
Observations:
- After increase in Epoch and batch size for SMOTE and Adam optimizer, the performance is slightly better than the previous model.
- Recall is 0.807987 for training and 0.796875 for validation dataset.
Model 12 - Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout¶
Lets add dropout to hidden layers
In [604]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [605]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train_over.shape[1]))
model.add(Dropout(0.4))
model.add(Dense(7,activation="relu"))
model.add(Dropout(0.2))
model.add(Dense(1,activation="sigmoid"))
In [606]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dropout (Dropout) │ (None, 14) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dropout_1 (Dropout) │ (None, 7) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [607]:
optimizer = tf.keras.optimizers.Adam() # defining Adam as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [608]:
start = time.time()
history = model.fit(X_train_over, y_train_over, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - loss: 0.6931 - val_loss: 0.6663 Epoch 2/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.6541 - val_loss: 0.6205 Epoch 3/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.6229 - val_loss: 0.5715 Epoch 4/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5930 - val_loss: 0.5499 Epoch 5/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5786 - val_loss: 0.5322 Epoch 6/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5630 - val_loss: 0.5111 Epoch 7/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5517 - val_loss: 0.4985 Epoch 8/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5407 - val_loss: 0.4968 Epoch 9/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5282 - val_loss: 0.4916 Epoch 10/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5284 - val_loss: 0.4896 Epoch 11/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5282 - val_loss: 0.4864 Epoch 12/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.5247 - val_loss: 0.4814 Epoch 13/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.5135 - val_loss: 0.4813 Epoch 14/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.5151 - val_loss: 0.4791 Epoch 15/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5119 - val_loss: 0.4737 Epoch 16/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5117 - val_loss: 0.4714 Epoch 17/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5052 - val_loss: 0.4658 Epoch 18/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5044 - val_loss: 0.4693 Epoch 19/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5037 - val_loss: 0.4678 Epoch 20/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4995 - val_loss: 0.4619 Epoch 21/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5047 - val_loss: 0.4638 Epoch 22/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5037 - val_loss: 0.4633 Epoch 23/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5056 - val_loss: 0.4641 Epoch 24/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5134 - val_loss: 0.4641 Epoch 25/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5007 - val_loss: 0.4681
In [609]:
print("Time taken in seconds ",end-start)
Time taken in seconds 18.299034357070923
In [610]:
plot(history,'loss')
In [611]:
model_12_train_perf = model_performance_classification(model, X_train_over, y_train_over)
model_12_train_perf
319/319 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
Out[611]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.785126 | 0.785126 | 0.785141 | 0.785123 |
In [612]:
model_12_val_perf = model_performance_classification(model, X_val, y_val)
model_12_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[612]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.773125 | 0.773125 | 0.830208 | 0.790041 |
Observations:
- Model hasnt performed well when compared to some of the other models.
- Recall is 0.78526 for training and 0.773125 for validation dataset.
Model 13 - Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout(dropout to only one hidden layer)¶
In [613]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [614]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train_over.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dropout(0.2))
model.add(Dense(1,activation="sigmoid"))
In [615]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dropout (Dropout) │ (None, 7) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [616]:
optimizer = tf.keras.optimizers.Adam() # defining Adam as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [617]:
start = time.time()
history = model.fit(X_train_over, y_train_over, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 2s 6ms/step - loss: 0.6841 - val_loss: 0.6321 Epoch 2/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.6061 - val_loss: 0.5592 Epoch 3/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.5610 - val_loss: 0.5269 Epoch 4/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5273 - val_loss: 0.5112 Epoch 5/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5078 - val_loss: 0.4911 Epoch 6/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4954 - val_loss: 0.4849 Epoch 7/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4835 - val_loss: 0.4791 Epoch 8/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4765 - val_loss: 0.4768 Epoch 9/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4694 - val_loss: 0.4718 Epoch 10/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4653 - val_loss: 0.4722 Epoch 11/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4623 - val_loss: 0.4665 Epoch 12/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4603 - val_loss: 0.4680 Epoch 13/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4581 - val_loss: 0.4675 Epoch 14/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4532 - val_loss: 0.4694 Epoch 15/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4468 - val_loss: 0.4615 Epoch 16/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4425 - val_loss: 0.4697 Epoch 17/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4465 - val_loss: 0.4679 Epoch 18/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4436 - val_loss: 0.4630 Epoch 19/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4415 - val_loss: 0.4663 Epoch 20/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4441 - val_loss: 0.4640 Epoch 21/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4379 - val_loss: 0.4540 Epoch 22/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4366 - val_loss: 0.4617 Epoch 23/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4399 - val_loss: 0.4699 Epoch 24/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4364 - val_loss: 0.4667 Epoch 25/25 160/160 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4339 - val_loss: 0.4598
In [618]:
print("Time taken in seconds ",end-start)
Time taken in seconds 17.973960876464844
In [619]:
plot(history,'loss')
In [620]:
model_13_train_perf = model_performance_classification(model, X_train_over, y_train_over)
model_13_train_perf
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[620]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.806417 | 0.806417 | 0.806473 | 0.806408 |
In [621]:
model_13_val_perf = model_performance_classification(model, X_val, y_val)
model_13_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
Out[621]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.786875 | 0.786875 | 0.836198 | 0.801777 |
Observations:
- Model with 1 dropouts is performing better than 2 dropouts.
- Recall is 0.806417 for training and 0.786875 for validation dataset.
- Pick Model with 1 dropouts and change the epoch and batch size
Model 14 - Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Dropout(dropout to only hidden layer), Change Epochs and Batch Size¶
In [622]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [623]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train_over.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dropout(0.2))
model.add(Dense(1,activation="sigmoid"))
In [624]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dropout (Dropout) │ (None, 7) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [625]:
optimizer = tf.keras.optimizers.Adam() # defining Adam as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [626]:
start = time.time()
history = model.fit(X_train_over, y_train_over, validation_data=(X_val,y_val) , batch_size=new_batch_size, epochs=new_epochs)
end=time.time()
Epoch 1/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - loss: 0.6587 - val_loss: 0.6112 Epoch 2/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6110 - val_loss: 0.5596 Epoch 3/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5732 - val_loss: 0.5366 Epoch 4/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5510 - val_loss: 0.5105 Epoch 5/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5295 - val_loss: 0.4853 Epoch 6/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5086 - val_loss: 0.4764 Epoch 7/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4960 - val_loss: 0.4627 Epoch 8/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4907 - val_loss: 0.4632 Epoch 9/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4842 - val_loss: 0.4626 Epoch 10/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4829 - val_loss: 0.4561 Epoch 11/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4806 - val_loss: 0.4577 Epoch 12/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step - loss: 0.4735 - val_loss: 0.4578 Epoch 13/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4720 - val_loss: 0.4546 Epoch 14/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4747 - val_loss: 0.4530 Epoch 15/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4686 - val_loss: 0.4500 Epoch 16/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4682 - val_loss: 0.4556 Epoch 17/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4668 - val_loss: 0.4529 Epoch 18/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4618 - val_loss: 0.4556 Epoch 19/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4626 - val_loss: 0.4471 Epoch 20/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4641 - val_loss: 0.4467 Epoch 21/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4588 - val_loss: 0.4495 Epoch 22/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4605 - val_loss: 0.4511 Epoch 23/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4572 - val_loss: 0.4495 Epoch 24/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4564 - val_loss: 0.4478 Epoch 25/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4531 - val_loss: 0.4485 Epoch 26/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4574 - val_loss: 0.4480 Epoch 27/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4529 - val_loss: 0.4442 Epoch 28/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4538 - val_loss: 0.4455 Epoch 29/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4506 - val_loss: 0.4474 Epoch 30/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4464 - val_loss: 0.4476 Epoch 31/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4514 - val_loss: 0.4459 Epoch 32/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4459 - val_loss: 0.4468 Epoch 33/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4487 - val_loss: 0.4446 Epoch 34/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4443 - val_loss: 0.4400 Epoch 35/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4449 - val_loss: 0.4430 Epoch 36/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4448 - val_loss: 0.4451 Epoch 37/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4455 - val_loss: 0.4455 Epoch 38/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4450 - val_loss: 0.4438 Epoch 39/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4466 - val_loss: 0.4444 Epoch 40/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4413 - val_loss: 0.4499 Epoch 41/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4434 - val_loss: 0.4432 Epoch 42/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4409 - val_loss: 0.4440 Epoch 43/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4382 - val_loss: 0.4424 Epoch 44/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4388 - val_loss: 0.4410 Epoch 45/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4420 - val_loss: 0.4439 Epoch 46/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4419 - val_loss: 0.4471 Epoch 47/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4421 - val_loss: 0.4504 Epoch 48/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4386 - val_loss: 0.4455 Epoch 49/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4403 - val_loss: 0.4472 Epoch 50/50 102/102 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4344 - val_loss: 0.4425
In [627]:
print("Time taken in seconds ",end-start)
Time taken in seconds 26.737415552139282
In [628]:
plot(history,'loss')
In [629]:
model_14_train_perf = model_performance_classification(model, X_train_over, y_train_over)
model_14_train_perf
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[629]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.807692 | 0.807692 | 0.807711 | 0.807689 |
In [630]:
model_14_val_perf = model_performance_classification(model, X_val, y_val)
model_14_val_perf
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
Out[630]:
| Accuracy | Recall | Precision | F1 Score | |
|---|---|---|---|---|
| 0 | 0.78 | 0.78 | 0.831569 | 0.795649 |
Observations:
- Not much of change in performance after increasing the epoch/batch size.
- Recall is 0.807692 for training and 0.78 for validation dataset.
- SMOTE Models did not perform well overall when compared to some of the other models.
Model Performance Comparison and Final Model Selection¶
In [631]:
# training performance comparison
models_train_comp_df = pd.concat(
[
model_0_train_perf.T,
model_1_train_perf.T,
model_2_train_perf.T,
model_3_train_perf.T,
model_4_train_perf.T,
model_5_train_perf.T,
model_6_train_perf.T,
model_7_train_perf.T,
model_8_train_perf.T,
model_9_train_perf.T,
model_10_train_perf.T,
model_11_train_perf.T,
model_12_train_perf.T,
model_13_train_perf.T,
model_14_train_perf.T
],
axis=1,
)
models_train_comp_df.columns = [
"Neural Network (SGD, without class weights)",
"Neural Network (SGD, with class weights)",
"Neural Network (SGD with Momentum, with class weights)",
"Neural Network (Adam , No Regularization)",
"Neural Network with Adam Optimizer(Change Epoch and Batch Size)",
"Neural Network (Adam, dropout [0.4,0.2])",
"Neural Network with Adam Optimizer and Dropout(Only in 1 hidden layer)",
"Neural Network with Adam Optimizer, Dropout(Only in 1 hidden layer), Change Epochs and Batch Size",
"Neural Network with Balanced Data (by applying SMOTE), SGD Optimizer",
"Neural Network with Balanced Data (by applying SMOTE),SGD Optimizer,Change Epoch and Batch Size",
"Neural Network with Balanced Data (by applying SMOTE) and Adam Optimizer",
"Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Change Epoch and Batch size",
"Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout",
"Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout(dropout to only 1 hidden layer)",
"Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Dropout(dropout to only 1 hidden layer), Change Epochs and Batch Size"
]
In [640]:
# Validation performance comparison
models_val_comp_df = pd.concat(
[
model_0_val_perf.T,
model_1_val_perf.T,
model_2_val_perf.T,
model_3_val_perf.T,
model_4_val_perf.T,
model_5_val_perf.T,
model_6_val_perf.T,
model_7_val_perf.T,
model_8_val_perf.T,
model_9_val_perf.T,
model_10_val_perf.T,
model_11_val_perf.T,
model_12_val_perf.T,
model_13_val_perf.T,
model_14_val_perf.T
],
axis=1,
)
models_val_comp_df.columns = [
"Neural Network (SGD, without class weights)",
"Neural Network (SGD, with class weights)",
"Neural Network (SGD with Momentum, with class weights)",
"Neural Network (Adam , No Regularization)",
"Neural Network with Adam Optimizer(Change Epoch and Batch Size)",
"Neural Network (Adam, dropout [0.4,0.2])",
"Neural Network with Adam Optimizer and Dropout(Only in 1 hidden layer)",
"Neural Network with Adam Optimizer, Dropout(Only in 1 hidden layer), Change Epochs and Batch Size",
"Neural Network with Balanced Data (by applying SMOTE), SGD Optimizer",
"Neural Network with Balanced Data (by applying SMOTE),SGD Optimizer,Change Epoch and Batch Size",
"Neural Network with Balanced Data (by applying SMOTE) and Adam Optimizer",
"Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Change Epoch and Batch size",
"Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout",
"Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout(dropout to only 1 hidden layer)",
"Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Dropout(dropout to only 1 hidden layer), Change Epochs and Batch Size"
]
In [633]:
models_train_comp_df
Out[633]:
| Neural Network (SGD, without class weights) | Neural Network (SGD, with class weights) | Neural Network (SGD with Momentum, with class weights) | Neural Network (Adam , No Regularization) | Neural Network with Adam Optimizer(Change Epoch and Batch Size) | Neural Network (Adam, dropout [0.4,0.2]) | Neural Network with Adam Optimizer and Dropout(Only in 1 hidden layer) | Neural Network with Adam Optimizer, Dropout(Only in 1 hidden layer), Change Epochs and Batch Size | Neural Network with Balanced Data (by applying SMOTE), SGD Optimizer | Neural Network with Balanced Data (by applying SMOTE),SGD Optimizer,Change Epoch and Batch Size | Neural Network with Balanced Data (by applying SMOTE) and Adam Optimizer | Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Change Epoch and Batch size | Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout | Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout(dropout to only 1 hidden layer) | Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Dropout(dropout to only 1 hidden layer), Change Epochs and Batch Size | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Accuracy | 0.796250 | 0.777500 | 0.812031 | 0.859688 | 0.859062 | 0.855313 | 0.855938 | 0.863906 | 0.769231 | 0.783458 | 0.806711 | 0.807987 | 0.785126 | 0.806417 | 0.807692 |
| Recall | 0.796250 | 0.777500 | 0.812031 | 0.859688 | 0.859062 | 0.855313 | 0.855938 | 0.863906 | 0.769231 | 0.783458 | 0.806711 | 0.807987 | 0.785126 | 0.806417 | 0.807692 |
| Precision | 0.634014 | 0.832367 | 0.845554 | 0.851531 | 0.849094 | 0.847971 | 0.849774 | 0.855526 | 0.769428 | 0.783461 | 0.806722 | 0.808011 | 0.785141 | 0.806473 | 0.807711 |
| F1 Score | 0.705931 | 0.793818 | 0.822852 | 0.843477 | 0.846678 | 0.835461 | 0.835303 | 0.850778 | 0.769188 | 0.783457 | 0.806709 | 0.807983 | 0.785123 | 0.806408 | 0.807689 |
In [641]:
models_val_comp_df
Out[641]:
| Neural Network (SGD, without class weights) | Neural Network (SGD, with class weights) | Neural Network (SGD with Momentum, with class weights) | Neural Network (Adam , No Regularization) | Neural Network with Adam Optimizer(Change Epoch and Batch Size) | Neural Network (Adam, dropout [0.4,0.2]) | Neural Network with Adam Optimizer and Dropout(Only in 1 hidden layer) | Neural Network with Adam Optimizer, Dropout(Only in 1 hidden layer), Change Epochs and Batch Size | Neural Network with Balanced Data (by applying SMOTE), SGD Optimizer | Neural Network with Balanced Data (by applying SMOTE),SGD Optimizer,Change Epoch and Batch Size | Neural Network with Balanced Data (by applying SMOTE) and Adam Optimizer | Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Change Epoch and Batch size | Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout | Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout(dropout to only 1 hidden layer) | Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, Dropout(dropout to only 1 hidden layer), Change Epochs and Batch Size | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Accuracy | 0.796250 | 0.781250 | 0.803125 | 0.847500 | 0.841875 | 0.843750 | 0.844375 | 0.850000 | 0.767500 | 0.781250 | 0.783125 | 0.796875 | 0.773125 | 0.786875 | 0.780000 |
| Recall | 0.796250 | 0.781250 | 0.803125 | 0.847500 | 0.841875 | 0.843750 | 0.844375 | 0.850000 | 0.767500 | 0.781250 | 0.783125 | 0.796875 | 0.773125 | 0.786875 | 0.780000 |
| Precision | 0.634014 | 0.833338 | 0.833083 | 0.836086 | 0.827511 | 0.832212 | 0.833886 | 0.837950 | 0.834171 | 0.832699 | 0.834075 | 0.836055 | 0.830208 | 0.836198 | 0.831569 |
| F1 Score | 0.705931 | 0.796914 | 0.813496 | 0.827564 | 0.826503 | 0.820695 | 0.820434 | 0.834656 | 0.786109 | 0.796810 | 0.798500 | 0.809459 | 0.790041 | 0.801777 | 0.795649 |
Observations:
Lets choose the top 5 models with higher recall score on validation dataset
- Neural Network with Adam Optimizer, Dropout(Only in 1 hidden layer), Change Epochs and Batch Size - Recall is 0.850000
- Neural Network (Adam , No Regularization) - Recall is 0.847500
- Neural Network with Adam Optimizer and Dropout(Only in 1 hidden layer) - Recall is 0.844375
- Neural Network (Adam, dropout [0.4,0.2]) - Recall is 0.843750
- Neural Network with Adam Optimizer(Change Epoch and Batch Size) - Recall is 0.841875
The top model is Neural Network with Adam Optimizer, Dropout(Only in 1 hidden layer), Change Epochs(50) and Batch Size(100) since it generalizes better when compared to other model. The Recall on validation dataset is 0.85 and on training dataset,its 0.863906.
Final Model
In [642]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [643]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train.shape[1]))
model.add(Dense(7,activation="relu"))
model.add(Dropout(0.2))
model.add(Dense(1,activation="sigmoid"))
In [644]:
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 14) │ 168 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 7) │ 105 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dropout (Dropout) │ (None, 7) │ 0 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 1) │ 8 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 281 (1.10 KB)
Trainable params: 281 (1.10 KB)
Non-trainable params: 0 (0.00 B)
In [645]:
optimizer = tf.keras.optimizers.Adam() # defining Adam as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer)
In [646]:
start = time.time()
history = model.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=new_batch_size, epochs=new_epochs)
end=time.time()
Epoch 1/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 2s 10ms/step - loss: 0.6390 - val_loss: 0.5463 Epoch 2/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - loss: 0.5345 - val_loss: 0.4759 Epoch 3/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4912 - val_loss: 0.4517 Epoch 4/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4752 - val_loss: 0.4396 Epoch 5/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4595 - val_loss: 0.4315 Epoch 6/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4533 - val_loss: 0.4251 Epoch 7/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4443 - val_loss: 0.4182 Epoch 8/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4328 - val_loss: 0.4111 Epoch 9/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4251 - val_loss: 0.4040 Epoch 10/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4218 - val_loss: 0.3965 Epoch 11/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4085 - val_loss: 0.3899 Epoch 12/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4059 - val_loss: 0.3831 Epoch 13/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3987 - val_loss: 0.3770 Epoch 14/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3925 - val_loss: 0.3727 Epoch 15/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3874 - val_loss: 0.3686 Epoch 16/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3790 - val_loss: 0.3656 Epoch 17/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3788 - val_loss: 0.3631 Epoch 18/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3686 - val_loss: 0.3601 Epoch 19/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3732 - val_loss: 0.3593 Epoch 20/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3707 - val_loss: 0.3585 Epoch 21/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3679 - val_loss: 0.3574 Epoch 22/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3650 - val_loss: 0.3571 Epoch 23/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3573 - val_loss: 0.3560 Epoch 24/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3618 - val_loss: 0.3553 Epoch 25/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3611 - val_loss: 0.3550 Epoch 26/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3581 - val_loss: 0.3546 Epoch 27/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3572 - val_loss: 0.3538 Epoch 28/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3591 - val_loss: 0.3538 Epoch 29/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3549 - val_loss: 0.3536 Epoch 30/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3550 - val_loss: 0.3535 Epoch 31/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3522 - val_loss: 0.3529 Epoch 32/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3502 - val_loss: 0.3532 Epoch 33/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3499 - val_loss: 0.3536 Epoch 34/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3488 - val_loss: 0.3535 Epoch 35/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3490 - val_loss: 0.3531 Epoch 36/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3499 - val_loss: 0.3534 Epoch 37/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3493 - val_loss: 0.3533 Epoch 38/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3491 - val_loss: 0.3532 Epoch 39/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step - loss: 0.3495 - val_loss: 0.3532 Epoch 40/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step - loss: 0.3481 - val_loss: 0.3534 Epoch 41/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - loss: 0.3511 - val_loss: 0.3531 Epoch 42/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.3470 - val_loss: 0.3530 Epoch 43/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - loss: 0.3461 - val_loss: 0.3535 Epoch 44/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3452 - val_loss: 0.3530 Epoch 45/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3465 - val_loss: 0.3538 Epoch 46/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3450 - val_loss: 0.3533 Epoch 47/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.3430 - val_loss: 0.3537 Epoch 48/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3410 - val_loss: 0.3541 Epoch 49/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3456 - val_loss: 0.3542 Epoch 50/50 64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.3425 - val_loss: 0.3540
In [648]:
y_train_pred = model.predict(X_train)
y_val_pred = model.predict(X_val)
y_test_pred = model.predict(X_test)
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step 50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step 63/63 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
In [649]:
print("Classification Report - Train data",end="\n\n")
cr = classification_report(y_train,y_train_pred>0.5)
print(cr)
Classification Report - Train data
precision recall f1-score support
0.0 0.88 0.97 0.92 5096
1.0 0.79 0.49 0.60 1304
accuracy 0.87 6400
macro avg 0.83 0.73 0.76 6400
weighted avg 0.86 0.87 0.86 6400
In [650]:
print("Classification Report - Validation data",end="\n\n")
cr = classification_report(y_val,y_val_pred>0.5)
print(cr)
Classification Report - Validation data
precision recall f1-score support
0.0 0.87 0.96 0.91 1274
1.0 0.73 0.42 0.53 326
accuracy 0.85 1600
macro avg 0.80 0.69 0.72 1600
weighted avg 0.84 0.85 0.83 1600
In [651]:
print("Classification Report - Test data",end="\n\n")
cr = classification_report(y_test,y_test_pred>0.5)
print(cr)
Classification Report - Test data
precision recall f1-score support
0.0 0.87 0.96 0.92 1593
1.0 0.76 0.45 0.57 407
accuracy 0.86 2000
macro avg 0.82 0.71 0.74 2000
weighted avg 0.85 0.86 0.85 2000
Observations:
- Recall(Weighted Avg) for the final model as follows
- Training - 0.87
- Validation - 0.85
- Testing - 0.86
We see that the model is able to generalize well with the test dataset.
Actionable Insights and Business Recommendations¶
- Recommend the the final model to the bank . This model helps reduce False Negatives.It has genarlized well with test data and has the highest recall score of 86% with test dataset.
- We saw in our analysis that customers who churned , the highest no. of customers in that category were not active members. Bank should reach out to customers who are not active with different schemes to keep the customer engaged with the bank.
- Banks should come up with new marketing strategies to attract customers to use more of their products thereby keeping the customers engaged with the bank.
- The bank is doing well in France and has the highest customers base. Banks in Germany and Spain can interact with bank location in France to see if they are doing things differently and learn from Germany location.
Power Ahead ___